User interface for displaying point clouds generated by a lidar device on a uav

ABSTRACT

Techniques are disclosed for real-time mapping in a movable object environment. A system for real-time mapping in a movable object environment, may include at least one movable object including a computing device, a scanning sensor electronically coupled to the computing device, and a positioning sensor electronically coupled to the computing device. The system may further include a client device in communication with the at least one movable object, the client device including a visualization application which is configured to receive point cloud data from the scanning sensor and position data from the positioning sensor, record the point cloud data and the position data to a storage location, generate a real-time visualization of the point cloud data and the position data as it is received, and display the real-time visualization using a user interface provided by the visualization application.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/211,509, filed Mar. 24, 2021, which is a continuation of U.S. patentapplication Ser. No. 16/664,743, filed Oct. 25, 2019 (now U.S. Pat. No.10,983,201), which claims the benefit of U.S. Provisional ApplicationNo. 62/752,273, filed Oct. 29, 2018, each of which is herebyincorporated by reference.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD OF THE INVENTION

The disclosed embodiments relate generally to techniques for mapping andmore particularly, but not exclusively, to a user interface fordisplaying point clouds generated by a LiDAR device on a UAV.

BACKGROUND

Movable objects such as unmanned aerial vehicles (UAVs) can be used forperforming surveillance, reconnaissance, and exploration tasks forvarious applications. Movable objects may carry a payload, includingvarious sensors, which enables the movable object to capture sensor dataduring movement of the movable objects. The captured sensor data may berendered on a client device, such as a client device in communicationwith the movable object via a remote control, remote server, or othercomputing device.

SUMMARY

Techniques are disclosed for real-time mapping in a movable objectenvironment. A system for real-time mapping in a movable objectenvironment, may include at least one movable object, wherein themovable object includes a computing device, a scanning sensor coupled tothe at least one movable object, and a positioning sensor coupled to theat least one movable object. The system may further include a clientdevice in communication with the at least one movable object, the clientdevice including at least one processor and a visualization application,the visualization application including instructions which, whenexecuted by the processor, cause the visualization application toreceive point cloud data from the scanning sensor and position data fromthe positioning sensor, record the point cloud data and the positiondata to a storage location, generate a real-time visualization of thepoint cloud data and the position data as it is received, and displaythe real-time visualization using a user interface provided by thevisualization application.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an example of a movable object in a movable objectenvironment, in accordance with various embodiments.

FIG. 2 illustrates an example of a movable object architecture in amovable object environment, in accordance with various embodiments.

FIG. 3 illustrates an example of a mapping manager in a movable objectenvironment, in accordance with various embodiments.

FIGS. 4A and 4B illustrate an example of a hierarchical data structure,in accordance with various embodiments.

FIGS. 5A and 5B illustrate an example of outlier removal in mappingdata, in accordance with various embodiments.

FIG. 6 illustrates an example of intensity values in mapping data, inaccordance with various embodiments.

FIG. 7 illustrates an example of supporting a movable object interfacein a software development environment, in accordance with variousembodiments.

FIG. 8 illustrates an example of a movable object interface, inaccordance with various embodiments.

FIG. 9 illustrates an example of components for a movable object in asoftware development kit (SDK), in accordance with various embodiments.

FIG. 10 shows a flowchart of a method of target mapping in a movableobject environment, in accordance with various embodiments.

FIG. 11 shows a graphical user interface (GUI) for real-time mapping ina movable object environment, in accordance with various embodiments.

FIG. 12 shows example GUI widgets for real-time mapping in a movableobject environment, in accordance with various embodiments.

FIG. 13 shows an example of movable object logs shown in the GUI forreal-time mapping in a movable object environment, in accordance withvarious embodiments.

FIG. 14 shows an example of LiDAR map rendering settings for the GUI forreal-time mapping in a movable object environment, in accordance withvarious embodiments.

FIG. 15 shows an example communication pipeline between a movable objectand a client application, in accordance with an embodiment.

FIG. 16 shows an example UI architecture, in accordance with anembodiment

FIG. 17 shows a flowchart of a method for real-time mapping in a movableobject environment, in accordance with various embodiments.

DETAILED DESCRIPTION

The invention is illustrated, by way of example and not by way oflimitation, in the figures of the accompanying drawings in which likereferences indicate similar elements. It should be noted that referencesto “an” or “one” or “some” embodiment(s) in this disclosure are notnecessarily to the same embodiment, and such references mean at leastone.

The following description of the invention describes target mappingusing a movable object. For simplicity of explanation, an unmannedaerial vehicle (UAV) is generally used as an example of a movableobject. It will be apparent to those skilled in the art that other typesof movable objects can be used without limitation.

Embodiments enable a movable object to map a target environment inreal-time using data collected from a positioning sensor and a scanningsensor. Alternative embodiments may take advantage of post-processing togenerate a map following completion of one or more data collectionmissions executed by one or more movable objects. For example, thevarious embodiments may utilize scan matching techniques for mapping acomplex target environment. Embodiments can be used to provideLiDAR-based real-time mapping for various applications, such asconstruction, surveying, target inspection, etc. Rather than collectingdata to be post-processed into a map representation of the target, a mapcan be constructed in real-time, enabling a version of the map to berendered on a client device as it is collected. Such live rendering mayenable the user to determine if any areas within the target environmenthave not been scanned by a scanning sensor electronically coupled to themovable object. Additionally, a high-density version of the map can begenerated during the mapping mission and downloaded upon return of themovable object. In various embodiments, a mapping manager may utilize aparallel computing architecture to perform the real-time mapping whilethe movable object is performing its mapping mission. In someembodiments, the mapping data may be output as a LiDAR Data ExchangeFile (LAS) which may be used by various tools to render the map of thetarget environment and/or use the mapping data for further processing,planning, etc. Metadata embedded in the LAS output file can facilitateintegration of the map with various third-party tools. In variousembodiments, the map may be output in various file formats depending onuser preferences.

In some embodiments, a mapping manager can receive mapping data from ascanning sensor (such as a LiDAR sensor or other sensor that provideshigh resolution scanning of a target environment), and positioning datafrom a positioning sensor (e.g., a global positioning system (GPS)module, real-time kinematic (RTK) module, an inertial measurement unit(IMU) module, or other positioning sensor). The mapping data can begeo-referenced using the positioning data and used to construct the mapof the target environment. Embodiments objectively geo-reference themapping data, enabling various target environments to be mappedregardless of environment complexity.

FIG. 1 illustrates an example of a movable object in a movable objectenvironment 100, in accordance with various embodiments. As shown inFIG. 1 , client device 110 in a movable object environment 100 cancommunicate with a movable object 104 via a communication link 106. Themovable object 104 can be an unmanned aircraft, an unmanned vehicle, ahandheld device, and/or a robot. The client device 110 can be a portablepersonal computing device, a smart phone, a remote control, a wearablecomputer, a virtual reality/augmented reality system, and/or a personalcomputer. Additionally, the client device 110 can include a remotecontroller 111 and communication system 120A, which is responsible forhandling the communication between the client device 110 and the movableobject 104 via communication system 120B. For example, the communicationbetween the client device 110 and the movable object 104 (e.g., anunmanned aircraft) can include uplink and downlink communication. Theuplink communication can be used for transmitting control signals, thedownlink communication can be used for transmitting media or videostream, mapping data collected scanning sensors, or other sensor datacollected by other sensors.

In accordance with various embodiments, the communication link 106 canbe (part of) a network, which is based on various wireless technologies,such as the WiFi, Bluetooth, 3G/4G, and other radio frequencytechnologies. Furthermore, the communication link 106 can be based onother computer network technologies, such as the internet technology, orany other wired or wireless networking technology. In some embodiments,the communication link 106 may be a non-network technology, includingdirect point-to-point connections such as universal serial bus (USB) oruniversal asynchronous receiver-transmitter (UART).

In various embodiments, movable object 104 in a movable objectenvironment 100 can include a payload assembly 122 and a payload, suchas a scanning sensor 124 (e.g., a LiDAR sensor). Although the movableobject 104 is described generally as an aircraft, this is not intendedto be limiting, and any suitable type of movable object can be used. Oneof skill in the art would appreciate that any of the embodimentsdescribed herein in the context of aircraft systems can be applied toany suitable movable object (e.g., a UAV). In some instances, thepayload may be provided on the movable object 104 without requiring thepayload assembly.

In accordance with various embodiments, the movable object 104 mayinclude one or more movement mechanisms 116 (e.g., propulsionmechanisms), a sensing system 118, and a communication system 120B. Themovement mechanisms 116 can include one or more of rotors, propellers,blades, engines, motors, wheels, axles, magnets, nozzles, animals, orhuman beings. For example, the movable object may have one or morepropulsion mechanisms. The movement mechanisms may all be of the sametype. Alternatively, the movement mechanisms can be different types ofmovement mechanisms. The movement mechanisms 116 can be mounted on themovable object 104 (or vice-versa), using any suitable means such as asupport element (e.g., a drive shaft). The movement mechanisms 116 canbe mounted on any suitable portion of the movable object 104, such onthe top, bottom, front, back, sides, or suitable combinations thereof.

In some embodiments, the movement mechanisms 116 can enable the movableobject 104 to take off vertically from a surface or land vertically on asurface without requiring any horizontal movement of the movable object104 (e.g., without traveling down a runway). Optionally, the movementmechanisms 116 can be operable to permit the movable object 104 to hoverin the air at a specified position and/or orientation. One or more ofthe movement mechanisms 116 may be controlled independently of the othermovement mechanisms, for example by an application executing on clientdevice 110, computing device 112, or other computing device incommunication with the movement mechanisms. Alternatively, the movementmechanisms 116 can be configured to be controlled simultaneously. Forexample, the movable object 104 can have multiple horizontally orientedrotors that can provide lift and/or thrust to the movable object. Themultiple horizontally oriented rotors can be actuated to providevertical takeoff, vertical landing, and hovering capabilities to themovable object 104. In some embodiments, one or more of the horizontallyoriented rotors may spin in a clockwise direction, while one or more ofthe horizontally oriented rotors may spin in a counterclockwisedirection. For example, the number of clockwise rotors may be equal tothe number of counterclockwise rotors. The rotation rate of each of thehorizontally oriented rotors can be varied independently in order tocontrol the lift and/or thrust produced by each rotor, and therebyadjust the spatial disposition, velocity, and/or acceleration of themovable object 104 (e.g., with respect to up to three degrees oftranslation and up to three degrees of rotation). As discussed furtherherein, a controller, such as flight controller 114, can send movementcommands to the movement mechanisms 116 to control the movement ofmovable object 104. These movement commands may be based on and/orderived from instructions received from client device 110, computingdevice 112, or other entity.

The sensing system 118 can include one or more sensors that may sensethe spatial disposition, velocity, and/or acceleration of the movableobject 104 (e.g., with respect to various degrees of translation andvarious degrees of rotation). The one or more sensors can include any ofthe sensors, including GPS sensors, motion sensors, inertial sensors,proximity sensors, or image sensors. The sensing data provided by thesensing system 118 can be used to control the spatial disposition,velocity, and/or orientation of the movable object 104 (e.g., using asuitable processing unit and/or control module). Alternatively, thesensing system 118 can be used to provide data regarding the environmentsurrounding the movable object, such as weather conditions, proximity topotential obstacles, location of geographical features, location ofmanmade structures, and the like.

The communication system 120B enables communication with client device110 via communication link 106, which may include various wired and/orwireless technologies as discussed above, and communication system 120A.The communication system 120A or 120B may include any number oftransmitters, receivers, and/or transceivers suitable for wirelesscommunication. The communication may be one-way communication, such thatdata can be transmitted in only one direction. For example, one-waycommunication may involve only the movable object 104 transmitting datato the client device 110, or vice-versa. The data may be transmittedfrom one or more transmitters of the communication system 120B of themovable object to one or more receivers of the communication system 120Aof the client device, or vice-versa. Alternatively, the communicationmay be two-way communication, such that data can be transmitted in bothdirections between the movable object 104 and the client device 110. Thetwo-way communication can involve transmitting data from one or moretransmitters of the communication system 120B of the movable object 104to one or more receivers of the communication system 120A of the clientdevice 110, and transmitting data from one or more transmitters of thecommunication system 120A of the client device 110 to one or morereceivers of the communication system 120B of the movable object 104.

In some embodiments, a client device 110 may communicate with a mappingmanager 126 installed on computing device 112 over a transparenttransmission channel of a communication link 106. The transparenttransmission channel can be provided through the flight controller ofthe movable object which allows the data to pass through unchanged(e.g., “transparent”) to the mapping manager or other application oncomputing device 112. In some embodiments, mapping manager 126 mayutilize a software development kit (SDK), application programminginterfaces (APIs), or other interfaces made available by the movableobject, computing device, scanning sensor 124, etc. In variousembodiments, the mapping manager may be implemented by one or moreprocessors on movable object 104 (e.g., flight controller 114 or otherprocessors), computing device 112, remote controller 111, client device110, or other computing device in communication with movable object 104.In some embodiments, mapping manager 126 may be implemented as anapplication executing on client device 110, computing device 112, orother computing device in communication with movable object 104.

In some embodiments, an application executing on client device 110 orcomputing device 112 can provide control data to one or more of themovable object 104, payload assembly 122, and payload 124 and receiveinformation from one or more of the movable object 104, payload assembly122, and payload 124 (e.g., position and/or motion information of themovable object, payload assembly or payload; data sensed by the payloadsuch as image data captured by a payload camera or mapping data capturedby a LiDAR sensor; and data generated from image data captured by thepayload camera or LiDAR data generated from mapping data captured by theLiDAR sensor).

In some embodiments, the control data may result in a modification ofthe location and/or orientation of the movable object (e.g., via controlof the movement mechanisms 116), or a movement of the payload withrespect to the movable object (e.g., via control of the payload assembly122). The control data from the application may result in control of thepayload, such as control of the operation of scanning sensor 124, acamera or other image capturing device (e.g., taking still or movingpictures, zooming in or out, turning on or off, switching imaging modes,changing image resolution, changing focus, changing depth of field,changing exposure time, changing viewing angle or field of view).

In some instances, the communications from the movable object, payloadassembly and/or payload may include information obtained from one ormore sensors (e.g., of the sensing system 118 or of the scanning sensor124 or other payload) and/or data generated based on the sensinginformation. The communications may include sensed information obtainedfrom one or more different types of sensors (e.g., GPS sensors, motionsensors, inertial sensors, proximity sensors, or image sensors). Suchinformation may pertain to the position (e.g., location, orientation),movement, or acceleration of the movable object, payload assembly,and/or payload. Such information from a payload may include datacaptured by the payload or a sensed state of the payload.

In some embodiments, computing device 112 can be added to the movableobject. The computing device can be powered by the movable object andcan include one or more processors, such as CPUs, GPUs, fieldprogrammable gate arrays (FPGAs), system on chip (SoC),application-specific integrated circuit (ASIC), or other processors. Thecomputing device can include an operating system (OS), such as Windows10®, Linux®, Unix®-based operating systems, or other OS. Missionprocessing can be offloaded from the flight controller 114 to thecomputing device 112. In various embodiments, the mapping manager 126can execute on the computing device 112, client device 110, payload 124,a remote server (not shown), or other computing device.

In some embodiments, mapping manager 126 can be used to provideLiDAR-based real-time mapping for various applications, such asconstruction, surveying, target inspection, etc. Rather than collectingdata to be post-processed into a map representation of the target, a mapcan be constructed in real-time, enabling a version of the map to berendered on client device 110 as it is collected. Such live renderingmay enable the user to determine if any areas within the targetenvironment have not been scanned by scanning sensor 124. Additionally,another version of the map may be downloaded and used upon return of themovable object. In various embodiments, the mapping manager 126 mayutilize a parallel computing architecture in computing device 112 toperform the real-time mapping. In some embodiments, the mapping manager126 may perform data compression to transform a dense map into a sparsemap to be rendered on client device 110. By way of compressing the densemap into the sparse map, the mapping manager 126 may be used to reducedata size required for transmission from the movable object 104 to theclient device 110, and thus, data transmission time and bandwidth aresaved for efficient real-time map rendering. In such embodiments, thelive rendering of the map may be a lower resolution or a compressed dataversion of the map (i.e., a sparse map) compared to the version obtainedfrom the movable object upon its return from scanning the targetenvironment (i.e., a dense map). In some embodiments, the map may beoutput as a LiDAR Data Exchange File (LAS) which may be used by varioustools to render the map of the target environment and/or use the mappingdata for further processing, planning, etc. Metadata embedded in the LASoutput file can facilitate integration of the map with variousthird-party tools. In various embodiments, the map may be output invarious file formats depending on user preferences.

Mapping manager 126 can receive mapping data from scanning sensor 124.As discussed, scanning sensor 124 may be a LiDAR sensor or other sensorthat provides high resolution scanning of a target environment. Themapping manager 126 may also receive positioning data from a positioningsensor (e.g., a GPS module, RTK module, or other positioning sensor). Insome embodiments, the positioning sensor may be part of functionalmodules 108, sensing system 118, or a separate module coupled to movableobject 104 which provides positioning data for the movable object. Themapping data can be geo-referenced using the positioning data and usedto construct the map of the target environment. Prior methods of 3Dmapping have relied on complex environments that are conducive toscan-matching. Unlike prior mapping systems, which require complexenvironments in order to use scan-matching to prepare the map,embodiments objectively geo-reference the mapping data. This allows forvarious target environments to be mapped regardless of environmentcomplexity.

Additional details of the movable object architecture are describedbelow with respect to FIG. 2 .

FIG. 2 illustrates an example 200 of a movable object architecture in amovable object environment, in accordance with various embodiments. Asshown in FIG. 2 , a movable object 104 can include a computing device112 and flight controller 114. The computing device 112 can connect tothe scanning sensor 124 via a high bandwidth connection, such asEthernet or universal serial bus (USB). The computing device 112 mayalso connect to a positioning sensor 202 over a low bandwidthconnection, such as universal asynchronous receiver-transmitter (UART).As discussed, the positioning sensor 202 may be included as a separatemodule (as shown in FIG. 2 ) or may be included as part of functionalmodules 108 or sensing system 118. Positioning sensor 202 may include aradio 204, such as a 4G, 5G, or other cellular or mobile network radio.The radio 204 may be used by RTK module 206 to enhance positioning datacollected by GPS module 208. Although a GPS module is shown in FIG. 2 ,any global navigation satellite service may be used, such as GLOSNASS,Galileo, BeiDou, etc. RTK module 206 can receive a reference signal froma reference station using radio 204 and provide a correction to thepositioning data provided by GPS module 208. Additionally, GPS module208 can output a clock signal, such as a pulse per second (1PPS) signal,to the scanning sensor 124. This allows for the scanning sensor and theGPS sensor to apply synchronized time stamps to their collected datausing the same clock signal.

In various embodiments, the computing device 112 can connect to one ormore high bandwidth components, such as one or more cameras, a stereovision module, or payload. The computing device 112 can connect to theflight controller 114 via UART and/or USB to send and receive data toand from the remote control via communication system 120B. In variousembodiments, the computing device 112 may include one or more CPUs,GPUs, field programmable gate arrays (FPGA), systems on chip (SoC), orother processor(s).

Flight controller 114 can connect to various functional modules 108,such as magnetometer 210, barometer 212, and inertial measurement unit(IMU) 214. In some embodiments, communication system 120B can connect tocomputing device 112 instead of, or in addition to, flight controller114. In some embodiments, sensor data collected by the one or morefunctional modules 108 and the positioning sensor 202 can be passed fromthe flight controller 114 to the computing device 112.

In some embodiments, flight controller 114 and computing device 112 canbe implemented as separate devices (e.g., separate processors onseparate circuit boards). Alternatively, one or more of the flightcontroller 114 and computing device 112 can be implemented as a singledevice, such as an SoC. In various embodiments, computing device 112 maybe removable from the movable object.

FIG. 3 illustrates an example 300 of a mapping manager 126 in a movableobject environment, in accordance with various embodiments. As shown inFIG. 3 , a mapping manager 126 may execute on one or more processors 302of computing device 112. The one or more processors 302 may includeCPUs, GPUs, FGPAs, SoCs, or other processors, and may be part of aparallel computing architecture implemented by computing device 112. Themapping manager 126 may include sensor interfaces 303, data preparationmodule 308, and map generator 316.

Sensor interfaces 303 can include a scanning sensor interface 304 and apositioning sensor interface 306. The sensor interfaces 303 may includehardware and/or software interfaces. The scanning sensor interface 304can receive data from the scanning sensor (e.g., a LiDAR or otherscanning sensor) and the positioning sensor interface 306 can receivedata from the positioning sensor (e.g., a GPS sensor, an RTK sensor, anIMU sensor, and/or other positioning sensors or a combination thereof).In various embodiments, the scanning sensor may produce mapping data ina point cloud format. The point cloud of the mapping data may be athree-dimensional representation of the target environment. In someembodiments, the point cloud of the mapping data may be converted to amatrix representation. The positioning data may include GPS coordinatesfor the movable object and, in some embodiments, may include roll,pitch, and yaw values associated with the movable object correspondingto each GPS coordinate. The roll, pitch, and yaw values may be obtainedfrom the positioning sensor, such as an inertial measurement unit (IMU),or other sensor. As discussed, the positioning data may be obtained froman RTK module, which corrects the GPS coordinates based on a correctionsignal received from a reference station. In some embodiments, the RTKmodule may produce a variance value associated with each outputcoordinate. The variance value may represent the accuracy of thecorresponding positioning data. For example, if the movable object isperforming sharp movements, the variance value may go up, indicatingless accurate positioning data has been collected. The variance valuemay also vary depending on atmospheric conditions, leading to differentaccuracies measured by the movable object depending on the particularconditions present when the data was collected.

The positioning sensor and scanning sensor may share clock circuitry.For example, the positioning sensor may include clock circuitry andoutput a clock signal to the scanning sensor. In some embodiments, aseparate clock circuit may output a clock signal to both the scanningsensor and the positioning sensor. As such, the positioning data and themapping data may be time-stamped using the shared clock signal.

In some embodiments, the positioning sensor and scanning sensor mayoutput data with differing delays. For example, the positioning sensorand the scanning sensor may not start generating data at the same time.As such, the positioning data and/or mapping data may be buffered toaccount for the delay. In some embodiments, a buffer size may be chosenbased on the delay between the output of each sensor. In someembodiments, mapping manager can receive the data from the positioningsensor and scanning sensor and output synchronized data using thetimestamps shared by the sensor data with respect to the shared clocksignal. This enables the positioning data and mapping data to besynchronized before further processing. Additionally, the frequency ofthe data obtained from each sensor may be different. For example, thescanning sensor may be producing data in the range of hundreds of kHz,while the positioning sensor may be producing data in the range ofhundreds of Hz. Accordingly, to ensure each point of the mapping datahas corresponding positioning data, upsampling module 310 caninterpolate the lower frequency data to match the higher frequency data.For example, assuming the positioning data is produced by thepositioning sensor at 100 Hz and the mapping data is produced by thescanning sensor (e.g., a LiDAR sensor) at 100 kHz, the positioning datamay be upsampled from 100 Hz to 100 kHz. Various upsampling techniquesmay be used to upsample the positioning data. For example, a linear fitalgorithm, such as least squares, may be used. In some embodiments,non-linear fit algorithms may be used to upsample the positioning data.Additionally, the roll, pitch, yaw values of the positioning data mayalso be interpolated to match the frequency of the mapping data. In someembodiments, the roll, pitch, and yaw values may be spherical linearinterpolated (SLERP) to match the number of points in the mapping data.The time stamps may likewise be interpolated to match the interpolatedpositioning data.

Once the positioning data has been upsampled and synchronized with themapping data by upsampling module 310, geo-reference module 312 canconvert the matrix representation of the mapping data from the frame ofreference (or the reference coordinate system) in which it was collected(e.g., scanner reference frame or scanner reference coordinate system)to a desired frame of reference (or a desired reference coordinatesystem). For example, the positioning data may be converted from thescanner reference frame to a north-east-down (NED) reference frame (or aNED coordinate system). The reference frame to which the positioningdata is converted may vary depending on the application of the map thatis being produced. For example, if the map is being used in surveying,it may be converted to the NED reference frame. For another example, ifthe map is being used for rendering motions such as flight simulation,it may be converted to the FlightGear coordinate system. Otherapplications of the map may effect conversions of the positioning datato different reference frames or different coordinate systems.

Each point in the point cloud of the mapping data is associated with aposition in the scanner reference frame that is determined relative tothe scanning sensor. The positioning data of the movable object,produced by the positioning sensor, may then be used to convert thisposition in the scanner reference frame to the output reference frame ina world coordinate system, such as a GPS coordinate system. For example,the position of the scanning sensor in the world coordinate system isknown based on the positioning data. In some embodiments, thepositioning sensor and the scanning module may be offset (e.g., due tobeing located at different positions on the movable object). In suchembodiments, a further correction factoring in this offset may be usedto convert from the scanner reference frame to the output referenceframe (e.g., each measured position in the positioning data may becorrected using the offset between the positioning sensor and thescanning sensor). For each point in the point cloud of the mapping data,the corresponding positioning data can be identified using the timestamp. The point can then be converted to the new reference frame. Insome embodiments, the scanner reference frame can be converted into ahorizontal reference frame using the interpolated roll, pitch, and yawvalues from the positioning data. Once the mapping data has beenconverted into the horizontal reference frame, it may be furtherconverted into a Cartesian frame or other output reference frame. Onceeach point has been converted, the result is a geo-referenced pointcloud, with each point in the point cloud now referenced to the worldcoordinate system. In some embodiments, the geo-referenced point cloudcan be provided to map generator 316 before performing outlier removalto remove outlier data from the geo-referenced point cloud.

After the geo-referenced point cloud has been produced, outlier removalmodule 314 can remove outlier data from the geo-referenced point cloud.In some embodiments, the geo-referenced point cloud may be downsampled,reducing the number of outliers in the data. Downsampling of this datamay be performed using voxels. In some embodiments, the points in eachvoxel may be averaged, and one or more averaged points may be output pervoxel. As such, outlier points will be removed from the data set in thecourse of averaging the points in each voxel. In various embodiments,the resolution of the voxels (e.g., the size of each voxel), may bearbitrarily defined. This allows for sparse and dense downsampled pointclouds to be produced. The resolution may be determined by the user, orby the mapping manager based on, e.g., available computing resources,user preferences, default values, or other application-specificinformation. For example, a lower resolution (e.g., larger voxel size)may be used to produce a sparse downsampled point cloud forvisualization on a client device or a mobile device. Additionally, oralternatively, outliers may be removed statistically. For example, thedistance from each point to its nearest neighbor may be determined andstatistically analyzed. If the distance from a point to its nearestneighbor is greater than a threshold value (e.g., a standard deviationof the nearest neighbor distances in the point cloud), then that pointmay be removed from the point cloud. In some embodiments, the outlierremoval technique may be selectable by the user or be automaticallyselected by the mapping manager. In some embodiments, outlier removalmay be disabled.

As discussed, the point cloud data may be a three-dimensionalrepresentation of the target environment. This 3D representation can bedivided into voxels (e.g., 3D pixels).

After statistical outlier removal, the resulting point cloud data can beprovided to map generator 316. In some embodiments, the map generator316 may include a dense map generator 318 and/or a sparse map generator320. In such embodiments, dense map generator 318 can produce ahigh-density map from the point cloud data received before outlierremoval, and sparse map generator 320 can produce a low-density map fromthe sparse downsampled point cloud data received after outlier removal.In other embodiments, dense map generator 318 and sparse map generator320 may produce a high-density map and a low-density map separately fromthe point cloud received both after outlier removal. In suchembodiments, each map generator may generate the output map using thesame process but may vary the size of the voxels to produce high-densityor low-density maps. In some embodiments, the low-density map can beused by a client device or a mobile device to provide real-timevisualization of the mapping data. The high-density map can be output asa LIDAR Data Exchange File (LAS) or other file type to be used withvarious mapping, planning, analysis, or other tools.

The map generator may use the point cloud data to perform aprobabilistic estimation of the position of points in the map. Forexample, the map generator may use a 3D mapping library, such as OctoMapto produce the output map. The map generator can divide the point clouddata into voxels. For each voxel, the map generator can determine howmany points are in the voxel and, based on the number of points and thevariance associated with each point, determine the probability that apoint is in that voxel. The probability may be compared to an occupancythreshold and, if the probability is greater than the occupancythreshold, a point may be represented in that voxel in the output map.In some embodiments, the probability that a given voxel is occupied canbe represented as:

${P\left( {n{❘z_{1:t}}} \right)} = \left\lbrack {1 + {\frac{1 - {P\left( {n{❘z_{t}}} \right)}}{P\left( {n{❘z_{t}}} \right)}\frac{1 - {P\left( {n{❘z_{1:{t - 1}}}} \right)}}{P\left( {n{❘z_{1:{t - 1}}}} \right)}\frac{P(n)}{1 - {P(n)}}}} \right\rbrack^{- 1}$

The probability P(n|z_(1:t)) of a node n being occupied is a function ofthe current measurement z₁, a prior probability P(n), and the previousestimate P (n|z_(1:t−1)). Additionally, P(n|z_(t)) represents theprobability that voxel n is occupied given the measurement z_(t). Thisprobability may be augmented to include the variance of each point, asmeasured by the positioning sensor, as represented by the followingequations:

${P(n)} = {{\frac{1}{2}{P_{x}\left( {x,\mu_{x},\sigma_{x}^{2}} \right)}{P_{y}\left( {y,\mu_{y},\sigma_{y}^{2}} \right)}{P_{z}\left( {z,\mu_{z},\sigma_{z}^{2}} \right)}} + \frac{1}{2}}$${P\left( {n,\mu,\sigma^{2}} \right)} = {\frac{1}{\sqrt{2\pi\sigma^{2}}}e^{- \frac{{({x - \mu})}^{2}}{2\sigma^{2}}}}$

In the equations above, P(n) represents the total probability that avoxel n is occupied. The use of ½ in the above equation isimplementation specific, such that the probability is mapped to a rangeof ½-1. This range may vary, depending on the particular implementationin use. In the above equations, the total probability is the product ofprobabilities calculated for the x, y, and z dimensions. The probabilityin each dimension may be determined based on the mean, μ, for each pointin that dimension, and the variance, σ², of each measurement in a givendimension, with x, y, and z corresponding to the coordinate values of agiven point. A large number of points near the mean point in a givenvoxel may increase the probability, while a more diffuse collection ofpoints in the voxel may lower the probability. Likewise, a largevariance associated with the data (e.g., indicating lower accuracyposition data has been collected) may lower the probability while alower variance may increase the probability. P(n, μ, σ²) represents theGaussian distribution for the voxel, given the mean and variance valuesof the points in that voxel.

If the total probability of a voxel being occupied is greater than athreshold occupancy value, then a point can be added to that voxel. Insome embodiments, all of the points in that voxel can be averaged, andthe resulting mean coordinate can be used as the location of the pointin that voxel. This improves the accuracy of the resulting map overalternative methods, such as using the center point of an occupied voxelas the point, which may result in skewed results depending on theresolution of the voxels. In various embodiments, the occupancythreshold can be set based on the amount of processing resourcesavailable and/or based on the acceptable amount of noise in the data fora given application. For example, the occupancy threshold can be set toa default value of 70%. A higher threshold can be set to reduce noise.Additionally, the occupancy threshold may be set depending on thequality of the data being collected. For example, data collected underone set of conditions may be high quality (e.g., low variance) and alower occupancy threshold can be set, while lower quality data maynecessitate a higher occupancy threshold.

The resulting map data, with one point in each occupied voxel, can thenbe output as an LAS file, or other file format. In some embodiments, thegeo-referenced point cloud data may be output without additionalprocessing (e.g., outlier removal). In some embodiments, each point inthe point cloud data may also be associated with an intensity value. Theintensity value may represent various features of the object beingscanned, such as elevation above a reference plane, materialcomposition, etc. The intensity value for each point in the output mapmay be an average of the intensity values measured for each point in themapping data collected by the scanning sensor (e.g., a LiDAR sensor).

FIGS. 4A and 4B illustrate an example of a hierarchical data structure,in accordance with various embodiments. As discussed above, and as shownin FIG. 4A, data representing a 3D environment 400 can be divided into aplurality of voxels. As shown in FIG. 4A, the target environment can bedivided into eight voxels, with each voxel being further divided intoeight sub-voxels, and each sub-voxel divided into eight further smallersub-voxels. Each voxel may represent a different volumetric portion ofthe 3D environment. The voxels may be subdivided until a smallest voxelsize is reached. The resulting 3D environment can be represented as ahierarchical data structure 402, where the root of the data structurerepresents the entire 3D environment, and each child node represents adifferent voxel in different hierarchy within the 3D environment.

FIGS. 5A and 5B illustrate an example of outlier removal in mappingdata, in accordance with various embodiments. As shown in FIG. 5A, whena target object is scanned, it may be represented as a plurality ofpoints, with those points clustered on different parts of the object,including surfaces (such as surface 501), edges (such as edge 503), andother portions of the target object in the target environment. Forsimplicity of depiction, these surfaces, edges, etc. are shown solid. Invarious regions 500A-500F of the data, there are additional outlierpoints. This may be most noticeable in regions of empty space, as shownin FIG. 5A. These points are diffuse, as compared to the more denselypacked points of the surfaces and edges of the target object. Outlierremoval may be used to eliminate or reduce the number of these points inthe data. As discussed above, the geo-referenced point cloud data may bedownsampled, reducing the number of outliers in the data. Additionally,or alternatively, outliers may be removed statistically. For example,the distance from each point to its nearest neighbor may be determinedand statistically analyzed. If the distance from a point to its nearestneighbor is greater than a threshold value (e.g., a standard deviationof the nearest neighbor distances in the point cloud), then that pointmay be removed from the point cloud. As shown in FIG. 5B, followingoutlier removal, the regions of the point cloud data 502A-502F have beenreduced, providing a cleaner 3D map.

FIG. 6 illustrates an example 600 of intensity values in mapping data,in accordance with various embodiments. As shown in FIG. 6 , one exampleof intensity values may be to represent elevation above a referenceplane. In this example, different elevation ranges may be assigned adifferent intensity value 602-606, as depicted here using greyscalecoloration. In various embodiments, intensity may be represented usingdifferent colors to represent different values or ranges of values.Additionally, intensity may be used to represent different materialsbeing scanned. For example, steel and concrete will absorb and reflectthe incident radiation produced by the scanning sensor differently,enabling the scanning sensor to identify different materials in use.Each material may be encoded as a different intensity value associatedwith each point and represented by a different color in the output map.Additionally, although the example shown in FIG. 6 shows three greyscalecolors representing different elevation ranges, in various embodiments,continuous gradients of colors may be used to represent continuouschanges in elevation value above a reference plane.

FIG. 7 illustrates an example of supporting a movable object interfacein a software development environment, in accordance with variousembodiments. As shown in FIG. 7 , a movable object interface 703 can beused for providing access to a movable object 701 in a softwaredevelopment environment 700, such as a software development kit (SDK)environment. In some embodiments, the movable object interface 703, mayrender a real-time map generated by the mapping manager and otherinterfacing components for receiving user input. The real-time map maybe rendered on a display of a client device or other computing device incommunication with the movable object. As used herein, the SDK can be anonboard SDK implemented on an onboard environment that is coupled to themovable object 701. The SDK can also be a mobile SDK implemented on anoff-board environment that is coupled to a client device or a mobiledevice. As discussed above, the mapping manager can be implemented usingan onboard SDK coupled to the movable object 701 or a mobile SDK coupledto a client device or a mobile device to enable applications to performreal-time mapping, as described herein.

Furthermore, the movable object 701 can include various functionalmodules A-C 711-713, and the movable object interface 703 can includedifferent interfacing components A-C 731-733. Each said interfacingcomponent A-C 731-733 in the movable object interface 703 corresponds toa module A-C 711-713 in the movable object 701. In some embodiments, theinterfacing components may be rendered on a user interface of a displayof a client device or other computing device in communication with themovable object. In such an example, the interfacing components, asrendered, may include selectable command buttons for receiving userinput/instructions to control corresponding functional modules of themovable object.

In accordance with various embodiments, the movable object interface 703can provide one or more callback functions for supporting a distributedcomputing model between the application and movable object 701.

The callback functions can be used by an application for confirmingwhether the movable object 701 has received the commands. Also, thecallback functions can be used by an application for receiving theexecution results. Thus, the application and the movable object 701 caninteract even though they are separated in space and in logic.

As shown in FIG. 7 , the interfacing components A-C 731-733 can beassociated with the listeners A-C 741-743. A listener A-C 741-743 caninform an interfacing component A-C 731-733 to use a correspondingcallback function to receive information from the related module(s).

Additionally, a data manager 702, which prepares data 720 for themovable object interface 703, can decouple and package the relatedfunctionalities of the movable object 701. The data manager 702 may beonboard, that is coupled to or located on the movable object 701, whichprepares the data 720 to be communicated to the movable object interface703 via communication between the movable object 701 and a client deviceor a mobile device. The data manager 702 may be off board, that iscoupled to or located on a client device or a mobile device, whichprepares data 720 for the movable object interface 703 via communicationwithin the client device or the mobile device. Also, the data manager702 can be used for managing the data exchange between the applicationsand the movable object 701. Thus, the application developer does notneed to be involved in the complex data exchanging process. In someembodiments, mapping manager 126 may be one implementation of datamanager 702. In such an embodiment, the mapping manager is used formanaging mapping data, including generating a map using mapping data andpositioning data and rendering the generated map for display based on adefault setting or a user selection.

For example, the onboard or mobile SDK can provide a series of callbackfunctions for communicating instant messages and for receiving theexecution results from a movable object. The onboard or mobile SDK canconfigure the life cycle for the callback functions in order to makesure that the information interchange is stable and completed. Forexample, the onboard or mobile SDK can establish connection between amovable object and an application on a smart phone (e.g. using anAndroid system or an iOS system). Following the life cycle of a smartphone system, the callback functions, such as the ones receivinginformation from the movable object, can take advantage of the patternsin the smart phone system and update the statements accordingly to thedifferent stages in the life cycle of the smart phone system. 70 j FIG.8 illustrates an example of a movable object interface, in accordancewith various embodiments. As shown in FIG. 8 , a movable objectinterface 803 can be rendered on a display of a client device or othercomputing devices representing statuses of different components of amovable object 801. Thus, the applications, e.g., APPs 804-806, in themovable object environment 800 can access and control the movable object801 via the movable object interface 803. As discussed, these apps mayinclude an inspection app 804, a viewing app 805, and a calibration app806.

For example, the movable object 801 can include various modules, such asa camera 811, a battery 812, a gimbal 813, and a flight controller 814.

Correspondently, the movable object interface 803 can include a cameracomponent 821, a battery component 822, a gimbal component 823, and aflight controller component 824 to be rendered on a computing device orother computing devices to receive user input/instructions by way ofusing the APPs 804-806.

Additionally, the movable object interface 803 can include a groundstation component 826, which is associated with the flight controllercomponent 824. The ground station component operates to perform one ormore flight control operations, which may require a high-levelprivilege.

FIG. 9 illustrates an example of components for a movable object in asoftware development kit (SDK), in accordance with various embodiments.The SDK 900 may be an onboard SDK implemented on an onboard mappingmanager or a mobile SDK implemented on mapping manager located on aclient device or a mobile device. The SDK 900 may correspond to all or aportion of the mapping manager described above or may be used toimplement the mapping manager as a standalone application. As shown inFIG. 9 , the drone class 901 in the SDK 900 is an aggregation of othercomponents 902-907 for a movable object (e.g., a drone). The drone class901, which have access to the other components 902-907, can exchangeinformation with the other components 902-907 and controls the othercomponents 902-907.

In accordance with various embodiments, an application may be accessibleto only one instance of the drone class 901. Alternatively, multipleinstances of the drone class 901 can present in an application.

In the SDK, an application can connect to the instance of the droneclass 901 in order to upload the controlling commands to the movableobject. For example, the SDK may include a function for establishing theconnection to the movable object. Also, the SDK can disconnect theconnection to the movable object using an end connection function. Afterconnecting to the movable object, the developer can have access to theother classes (e.g. the camera class 902, the battery class 903, thegimbal class 904, and the flight controller class 905). Then, the droneclass 901 can be used for invoking the specific functions, e.g.providing access data which can be used by the flight controller tocontrol the behavior, and/or limit the movement, of the movable object.

In accordance with various embodiments, an application can use a batteryclass 903 for controlling the power source of a movable object. Also,the application can use the battery class 903 for planning and testingthe schedule for various flight tasks. As battery is one of the mostrestricted elements in a movable object, the application may seriouslyconsider the status of battery not only for the safety of the movableobject but also for making sure that the movable object can finish thedesignated tasks. For example, the battery class 903 can be configuredsuch that if the battery level is low, the movable object can terminatethe tasks and go home outright. For example, if the movable object isdetermined to have a battery level that is below a threshold level, thebattery class may cause the movable object to enter a power savingsmode. In power savings mode, the battery class may shut off, or reduce,power available to various components that are not integral to safelyreturning the movable object to its home. For example, cameras that arenot used for navigation and other accessories may lose power, toincrease the amount of power available to the flight controller, motors,navigation system, and any other systems needed to return the movableobject home, make a safe landing, etc.

Using the SDK, the application can obtain the current status andinformation of the battery by invoking a function to request informationfrom in the Drone Battery Class. In some embodiments, the SDK caninclude a function for controlling the frequency of such feedback.

In accordance with various embodiments, an application can use a cameraclass 902 for defining various operations on the camera in a movableobject, such as an unmanned aircraft. For example, in SDK, the CameraClass includes functions for receiving media data in SD card, getting &setting photo parameters, taking photo and recording videos.

An application can use the camera class 902 for modifying the setting ofphotos and records. For example, the SDK may include a function thatenables the developer to adjust the size of photos taken. Also, anapplication can use a media class for maintaining the photos andrecords.

In accordance with various embodiments, an application can use a gimbalclass 904 for controlling the view of the movable object. For example,the Gimbal Class can be used for configuring an actual view, e.g.setting a first personal view of the movable object. Also, the GimbalClass can be used for automatically stabilizing the gimbal, in order tobe focused on one direction. Also, the application can use the GimbalClass to change the angle of view for detecting different objects.

In accordance with various embodiments, an application can use a flightcontroller class 905 for providing various flight control informationand status about the movable object. As discussed, the flight controllerclass can include functions for receiving and/or requesting access datato be used to control the movement of the movable object across variousregions in a movable object environment.

Using the Flight Controller Class, an application can monitor the flightstatus, e.g. using instant messages. For example, the callback functionin the Flight Controller Class can send back the instant message everyone thousand milliseconds (1000 ms).

Furthermore, the Flight Controller Class allows a user of theapplication to investigate the instant message received from the movableobject. For example, the pilots can analyze the data for each flight inorder to further improve their flying skills.

In accordance with various embodiments, an application can use a groundstation class 907 to perform a series of operations for controlling themovable object.

For example, the SDK may require applications to have an SDK-LEVEL-2 keyfor using the Ground Station Class. The Ground Station Class can provideone-key-fly, on-key-go-home, manually controlling the drone by app (i.e.joystick mode), setting up a cruise and/or waypoints, and various othertask scheduling functionalities.

In accordance with various embodiments, an application can use acommunication component for establishing the network connection betweenthe application and the movable object.

FIG. 10 shows a flowchart of a method of target mapping in a movableobject environment, in accordance with various embodiments. Atoperation/step 1002, mapping data can be obtained from a scanning sensor(e.g., a LiDAR sensor) supported by a movable object (e.g., a UAV). Insome embodiments, the scanning sensor can be a LiDAR sensor. Atoperation/step 1004, positioning data can be obtained from a positioningsensor (e.g., a GPS sensor, an RTK sensor, an IMU sensor, and/or otherpositioning sensors or a combination thereof) supported by the movableobject (e.g., a UAV). In some embodiments, the positioning sensor can bean RTK sensor or an IMU sensor.

At operation/step 1006, the mapping data can be associated with thepositioning data based at least on time data associated with the mappingdata and the positioning data. In some embodiments, associating themapping data with the positioning data may include upsampling thepositioning data to include a number of positions equal to a number ofpoints in the mapping data, and associating each point in the mappingdata with a corresponding position in the upsampled positioning data. Insome embodiments, the time data associated with the mapping data and thepositioning data may be obtained using clock circuitry providing areference clock signal electronically coupled to the scanning sensor andthe positioning sensor.

At operation/step 1008, a map in a first coordinate system may begenerated based at least on the associated mapping data and positioningdata. In some embodiments, generating the map may include, for eachvoxel of a plurality of voxels of the map, determining one or morepoints from the mapping data to be located in the voxel, and determiningan occupancy probability for the voxel based at least on a number ofpoints in that voxel. In some embodiments, the occupancy probability isdetermined based on a variance of the positioning data associated withthe one or more points located in the voxel. In some embodiments, foreach voxel having an occupancy probability greater than an occupancyprobability threshold value, an average position of the one or morepoints in the voxel can be calculated, and a point can be generated inthe map at the average position. In some embodiments, for each voxelhaving an occupancy probability greater than the occupancy probabilitythreshold value, an average intensity value of the one or more points inthe voxel can be calculated, and the average intensity value can beassociated with the generated point in the map. In an embodiment, theaverage intensity value is calculated based on feature of each point inthe voxel, where the feature of each point is associated with anelevation or a material detected by the scanning sensor.

In some embodiments, the method may further include determining adistribution of points in the mapping data, each of the points in themapping data associated with a distance from a nearest neighboring pointin the mapping data, and removing any points associated with a distancegreater than a distance threshold value. In some embodiments, the methodmay further include downsampling the mapping data by a scaling factor,dividing the mapping data into a plurality of voxels, and outputting anaverage point from the downsampled mapping data for each of theplurality of voxels. In some embodiments, the method may further includetransforming the map into a second coordinate system and outputting thetransformed map. For example, the positioning data may be converted fromthe scanner reference frame to a north-east-down (NED) reference frame(or a NED coordinate system). The reference frame to which thepositioning data is converted may vary depending on the application ofthe map that is being produced. For example, if the map is being used insurveying, it may be converted to the NED reference frame. For anotherexample, if the map is being used for rendering motions such as flightsimulation, it may be converted to the FlightGear coordinate system.Other applications of the map may effect conversions of the positioningdata to different reference frames or different coordinate systems.

In some embodiments, geo-referencing as described above may be combinedwith scan matching, such as Simultaneous Localization and Mapping (SLAM)or LiDAR Odometry and Mapping (LOAM). Traditional methods make use ofSLAM with or without inertial navigation input. For example, somemethods inject IMU information with SLAM and sometimes inject odometryvia GPS which provides an improved mapping algorithm. Unlike traditionalmethods, embodiments can perform direct geo-referencing as discussedabove, and then a layer of SLAM or LOAM can be added on top of thegeo-references. This provides a robust mapping algorithm as thegeo-references serves as a floor on the quality of the resulting map.

In some embodiments, geo-referencing as described above may be combinedwith normal distribution transformation (NDT). NDT is a LiDAR scanregistration method which is in between a feature-based registrationmethod (such as LOAM) and a point-based registration method (such asiterative closest point). The “features” of the world are described bymultivariate Gaussian distributions defined in each voxel. A probabilitydensity function (PDF) is generated for each cell and points are matchedto the map by maximizing the sum of probability generated by the PDF,points x, and a transformation T:

$T = {\arg\min\limits_{T}{\sum\limits_{i = 1}^{k}{- {p_{i}\left( {Tx_{i}} \right)}}}}$$\begin{matrix}{{p_{i}\left( x_{i} \right)} \sim {N\left( {\overset{¯}{\mu_{i}},\Sigma_{i}} \right)}} & {{\overset{¯}{\mu}}_{i} = \begin{bmatrix}\mu_{x} \\\mu_{y} \\\mu_{z}\end{bmatrix}}\end{matrix}$

FIG. 11 shows a graphical user interface (GUI) 1100 for real-timemapping in a movable object environment, in accordance with variousembodiments. The user interface (UI) 1100 may be rendered on a display(e.g., a touchscreen, a light-emitting diode (LED) display, an organiclight-emitting diode (OLED) display, a monitor, etc.) of a clientdevice, such as a smartphone, tablet, laptop, desktop, or othercomputing device. In some embodiments, the client device may be clientdevice 110 discussed above. Additionally, or alternatively, the userinterface 1100 may be rendered on a display of a different client devicein communication with the movable object.

In some embodiments, user interface 1100 may be divided into multiplesub-parts for rendering digital objects in different areas of thedisplay. These sub-parts may include separate frames, panels, or otherelements that divide where information and/or functional elements aredisplayed in the user interface 1100. For example, the UI may include amain portion 1102 in which a real-time local map (e.g., generated byassociating mapping data and positioning data using time data) isrendered. In the example shown in FIG. 11 , an area may be scanned by amovable object using a LiDAR sensor. As discussed above, mapping data,such as point cloud data, collected by the LiDAR sensor may begeoreferenced based on positioning data obtained from a positioningsensor carried by the movable object. The resulting mapping data maythen be downsampled such that it can be rendered on the display of themobile device in real-time, as the mapping data is collected. Thedownsampled mapping data, when rendered, shows a real-time local mapwhich includes a representation of the area that has been scanned andobjects located therein, such as the trees shown in the example of FIG.11 . In some embodiments, the client application provides intuitivegesture controls so the user can rotate, zoom, and pan over therendering of the point cloud data via the user interface. For example,the client device may include a touchscreen interface, which enables theuser to interact with the real-time map and other user interfaceelements.

In various embodiments, other sub-parts may be overlaid on the mainportion 1102. For example, a first sub-part of the UI may include one ormore selectable commands 1104, while a second sub-part 1106 may includea global map showing the position of the movable object in a globalpositioning system and another sub-part 1108 may show currentinformation about the movable object, such as its status, any servicecodes, and its current position in X, Y, Z coordinates. Another sub-part1110 may include a representation of a FOV or an orientation of theLiDAR sensor, while other sub-parts may include a taskbar showing statusof each component (battery level, internet connection, etc.) of themovable object, etc. In some embodiments, the various sub-parts may berendered separately such that no sub-part is overlaid on any othersub-part.

In various embodiments, a user can view and/or control the collection ofmapping data using the user interface 1100. For example, the UI 1100 caninclude selectable commands 1104 that allow the user to send commands tocontrol the movable object. As shown in FIG. 11 , these selectablecommands 1104 may include: Start/Pause/Reset mapping, write mappingresults into a LiDAR data exchange (LAS) file, allow the user to set aname for the LAS file, Start/Stop recording of a file (e.g., such as arobot operating system (ROS) bag file) for replaying data collectionlater, or obtain device information regarding different components ofthe movable object. Additionally, the UI 1100 may include playbackcontrols, as shown in sub-part 1112, which may enable the user to pauseor speed up the playback of previously recorded mapping data, such as alocal file or a file stored remotely that is accessible to theapplication over a communication network. The playback controls caninclude controls for playback speed. For example, the user can play themapping data at 1× speed, or at various other speeds such as 0.5×, 2×,etc.

FIG. 12 shows additional user interface elements 1200 in accordance withan embodiment. In addition to the selectable commands 1104, the userinterface can enable the user to select storage where to write LAS file,as shown at 1202, and may provide a mapping status 1204. The mappingstatus 1204 may provide a status window of the scanning sensor,positioning sensor, onboard computer, and/or other components of themovable object to indicate if the movable object is ready to perform amapping mission. The background color of the status window may change incolor, such as change from red to yellow to green, where green indicatesthat the user can start mapping. Tapping on the status window provides amore detailed message of the mapping status. In some embodiments, theuser interface elements 1200 may be rendered as pop-up windows shown ona display of the client device. One or more of the user interfaceelements 1200 may pop-up automatically (e.g., when a stop mappingcommand is detected, a window 1202 may pop-up enabling the user toselect where to write and store the LAS file or name the LAS file) ormanually (e.g., when the user clicks on the write button in theselectable commands 1104 the window 1202 may then pop-up). Similarly,the mapping status element 1204 may pop-up automatically and may bedismissed automatically once the movable object is ready to perform amapping mission, or may pop-up in response to a user selecting the startbutton from selectable commands 1104 before the movable object is readyto perform a mapping mission.

FIG. 13 shows an example of movable object logs shown in the GUI forreal-time mapping in a movable object environment, in accordance withvarious embodiments. As shown in FIG. 13 , logs 1300 may be displayed tothe user via the user interface 1100. The logs 1300 may be displayed ina pop-up window, in a particular sub-part of the UI 1100 as a UIelement, or otherwise presented to the user through the user interface.The logs 1300 may record an entry whenever a user accesses theapplication via the user interface 1100. In some embodiments, the logs1300 may also include details about the movable object or componentsthereof prior to, during, and/or after a mapping mission has beenperformed. For example, the logs 1300 may present sensor performance ata particular time, as shown at 1302 (e.g., throughput of the LiDARsensor), commands that have been sent 1304 and received 1306, along withtimestamps associated with the sending and receiving of those commands,device status 1308, and/or other information associated with the movableobject, the user's interaction with the UI 1100, such as a size of thegenerated mapping data, etc. In some embodiments, logs can be filteredor sorted in order by date, timestamp, type, tag, or message. In someembodiments, the user may search for logs by tag identifier 1310 and/ormessage 1312. At least when sorted by date or timestamp, as shown inFIG. 13 , the user may be able to select a date or timestamp 1314 andthen view logs 1316 associated with that date or timestamp. In someembodiments, logs can be downloaded and shared with other apps.

FIG. 14 shows an example of LiDAR map rendering settings 1400 for theGUI for real-time mapping in a movable object environment, in accordancewith various embodiments. As discussed above, the client application canrender mapping data collected by the movable object through userinterface 1100. As shown in FIG. 14 , LiDAR Map Rendering Settings 1400can be modified by the user. The rendering settings can be used tocontrol how the mapping data is viewed. For example, the user may selectwhether to view the mapping data via a perspective or orthogonal view1402. In some embodiments, the rendering settings may also includeintensity coloring settings 1404. As discussed, mapping data may includeintensity values that can be used when rendering the mapping data toprovide a visual difference between points in the mapping data based ontheir intensity values. These intensity values may be used to representdifferent characteristics of the objects and/or area being scanned bythe LiDAR sensor. For example, as discussed above with respect to FIG. 6, the intensity values may represent elevation above a reference plane.In various embodiments, intensity may be represented using differentcolors to represent different values or ranges of values. Additionally,intensity may be used to represent different materials being scanned.For example, steel and concrete will absorb and reflect the incidentradiation produced by the scanning sensor differently, enabling thescanning sensor to identify different materials in use. Each materialmay be encoded as a different intensity value associated with each pointand represented by a different color in the output map.

In some embodiments, the user may also control the resolution 1406 ofthe mapping data. The resolution may be determined by the user, or bythe mapping manager based on, e.g., available computing resources, userpreferences, default values, or other application-specific information.For example, a lower resolution (e.g., larger voxel size) may be used toproduce a sparse downsampled point cloud for visualization on a clientdevice or a mobile device. In some embodiments, the user may specify aparticular resolution by voxel size through the map rendering settings1400 and/or may choose to increase or decrease the resolution inincrements. Such relative controls enable a user to change theresolution intuitively, without requiring the user to know how large orsmall to make the voxels in order to obtain their desired resolution. Insome embodiments, the mapping manager may cause the mapping data to berendered at a default resolution and the user may choose to increase ordecrease relative to the default resolution.

In some embodiments, the map rendering settings 1400 may also enable theuser to specify a source 1408 of the point cloud data. For example, themap rendering settings 1400 may include a drop down menu 1410 thatenables the user to select from a variety of sources, such as widebandprocessing of point cloud data received from the movable object in realtime, or ROS processing of point cloud data sent from a server or otherdata storage (e.g., used for debugging), etc. In some embodiments, theuser can specify a location of a file to be loaded, such as a file fromthe client device or from a server or other computing device incommunication with the client application (e.g., over one or morenetworks). In some embodiments, the map rendering settings 1400 may alsoinclude a selectable command 1412 to clear the currently rendered pointcloud data. For example, prior to loading new mapping data, the currentmapping data may be cleared by selecting command 1412.

FIG. 15 shows an example 1500 pipeline of communication between themovable object 104 and the application on client device 110. In someembodiments, as discussed, the movable object 104 can include an onboardcomputer which implements an onboard SDK 1502. As discussed, the mappingmanager 126 may be implemented by the onboard SDK 1502 and may includesensor interfaces, such as onboard sensor interface 1504 and widebandsensor interface 1506. Onboard sensor interface 1504 may be used toobtain sensor data and/or send commands to various sensors on movableobject 104, such as functional modules 108 which communicate via flightcontroller 114 and/or positioning sensor 202. Wideband sensor interface1506 may be used to obtain sensor data and/or send commands to varioushigh bandwidth sensors, such as a LiDAR sensor or other scanning sensor124. Other sensors which provide information to, or receive informationfrom, client application 1526 may include a command sensor to send andreceive specific commands (e.g., based on user input through selectablecommands 1104); a status sensor which can receive the current status,service and position covariance of the GNSS IMU sensor (e.g., which mayprovide the status data as shown in the status widget shown at 1204 inFIG. 12 ); a LiDAR Map Sensor which receives point cloud data and isused by the LiDAR map kit to render the point cloud data; a node statussensor which receives the status of different services running on theonboard device (e.g., which may be used to generate logs, as shown inFIG. 13 ).

In some embodiments, the client device 110 may include a mobile SDK1508, which can communicate with the onboard SDK 1502, and userinterface 1100. The mobile SDK 1508 may include a flight controllermanager 1510 which can communicate commands/data to, and data from, theflight controller via the onboard SDK 1502. For example, telemetry datamay be received from the flight controller, and commands to change thecourse of the movable object may be sent to the flight controller byflight controller manager 1510. In some embodiments, a flight controllercommunication manager 1512 can convert commands received from the userinto a machine-readable format that is understandable by the flightcontroller. This may include using a protocol library 1514 to encode thecommands in a format that is transmittable to, and understandable by,the flight controller. In some embodiments, the FC communication manager1512 and protocol library 1514 can similarly be used to decode datareceived from the flight controller to be rendered to the user via userinterface 1100. Similarly, a payload data manager 1516 may receivesensor data obtained by one or more high bandwidth sensors, such as ascanning sensor, stereoscopic camera, or other high bandwidth sensor,via wideband sensor interface 1506. In some embodiments, the payloaddata manager 1516 can also send commands to one or more wideband sensorsvia wideband sensor interface 1506. For example, the angle at which theLiDAR sensor is positioned may be adjustable in response to a commandfrom the user. The command can be packaged to be sent to the payload,such as a LiDAR sensor, by wideband communication manager 1518, usingprotocol library 1514, and then sent to the payload by payload datamanager 1516 via wideband sensor interface 1506.

When data is received from the sensors of the movable object, this datacan be received by UI communication manager 1520 and provided to a UXSDK 1524 of user interface 1100. The UX SDK 1524 can manage how variousUI elements are rendered in response to the sensor data obtained fromthe UI communication manager 1520. For example, the rendering of abattery icon may vary depending on the current battery level obtainedfrom the UI communication manager as well as the type of data receivedfrom the battery sensor (e.g., the number of battery cells in the batterin use by the movable object, etc.). Additionally, the UX SDK 1524 canmanage to request sensor data from various components of the movableobject periodically or in response to a specific request for sensor datareceived from the user via user interface 1100. The UX SDK 1524 can alsoconvert command instructions received from the user through the userinterface into commands and/or requests to be sent to the movableobject. The commands and/or requests can be sent by the UX SDK to the UIcommunication manager 1520. The UI communication manager 1520 canidentify the sensor or component of the movable object to which thecommand or request is being sent and route the command or request viathe FC communication manager 1512 or wideband communication manager1518. Responses to the commands (or other data generated in response tothe commands) can be similarly routed back to the user interface via theFC communication manager 1512 or wideband communication manager 1518 andthe UI communication manager 1520. In some embodiments, the UX SDK 1524can use LiDAR map kit 1522 to render point cloud data received from aLiDAR sensor. In some embodiments, the LiDAR map kit 1522 can implementa rendering framework and/or one or more libraries for rendering pointcloud data in a graphical user interface on behalf of the clientapplication.

FIG. 16 shows an example UI architecture 1600, in accordance with anembodiment. The UI architecture 1600 can include a UI communicationmanager 1520. As discussed above, the UI communication manager enablesthe application to communicate with (e.g., receive sensor data fromand/or provide commands to) the onboard and wideband sensors of themovable object. The UI architecture may also include a UX SDK 1524 thatincludes various UI widgets 1602. As discussed, the user interface mayprovide a variety of user interface elements, including selectablecommands 1104, a status widget 1204, mapping widgets 1106, 1108, 1110, atask bar and its associated widgets (e.g., related to battery level,communication strength, etc.), log interface 1300, map renderingsettings widget 1400, etc. The rendering of each of these widgets may bemanaged by UX SDK 1524. Additionally, a Lidar Map Widget 1604 providesaccess to the LiDAR map kit 1522 which handles point cloud rendering inthe user interface of the application.

FIG. 17 shows a flowchart of a method for real-time mapping in a movableobject environment, in accordance with various embodiments. Atoperation/step 1702, mapping data such as point cloud data is receivedfrom a scanning sensor and position data is received from a positioningsensor. At operation/step 1704, the point cloud data and the positiondata are recorded to a storage location.

At operation/step 1706, a real-time visualization of the point clouddata and the position data is generated as it is received. In someembodiments, generating the real-time visualization may further includegenerating the real-time visualization of the point cloud data based onan intensity value associated with each point in the point cloud data.The intensity value represents a material composition or height above areference plane associated with each point in the point cloud data. Atoperation/step 1708, the real-time visualization is displayed using auser interface provided by a visualization application.

In some embodiments, the method may further include receiving aselection of a mapping file, the mapping file including previouslycollected sensor data, the sensor data including historical point clouddata and historical position data, receiving a request to view playbackof the mapping file, and displaying a historical visualization generatedbased on the mapping file using the user interface.

In some embodiments, the method may further include receiving a requestto change a resolution of the point cloud data, changing a voxel sizebased at least on the request to generate a second point cloud datahaving a different resolution than the point cloud data, and generatingthe real-time visualization of the second point cloud data. In someembodiments. the resolution of the point cloud data is automatically setto a default resolution determined based at least on computing resourcesassociated with a client device.

In some embodiments, the method may further include obtaining sensordata from one or more sensors of a movable object, generating arepresentation of the sensor data using at least one associatedgraphical user interface element, and displaying the representation ofthe sensor data as an overlay on the real-time visualization.

Many features can be performed in, using, or with the assistance ofhardware, software, firmware, or combinations thereof. Consequently,features may be implemented using a processing system (e.g., includingone or more processors). Exemplary processors can include, withoutlimitation, one or more general purpose microprocessors (for example,single or multi-core processors), application-specific integratedcircuits, application-specific instruction-set processors, graphicsprocessing units, physics processing units, digital signal processingunits, coprocessors, network processing units, audio processing units,encryption processing units, and the like.

Features can be implemented in, using, or with the assistance of acomputer program product which is a storage medium (media) or computerreadable medium (media) having instructions stored thereon/in which canbe used to program a processing system to perform any of the featurespresented herein. The storage medium can include, but is not limited to,any type of disk including floppy disks, optical discs, DVD, CD-ROMs,microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs,DRAMs, VRAMs, flash memory devices, magnetic or optical cards,nanosystems (including molecular memory ICs), or any type of media ordevice suitable for storing instructions and/or data.

Stored on any one of the machine readable medium (media), features canbe incorporated in software and/or firmware for controlling the hardwareof a processing system, and for enabling a processing system to interactwith other mechanism utilizing the results. Such software or firmwaremay include, but is not limited to, application code, device drivers,operating systems and execution environments/containers.

Features of the invention may also be implemented in hardware using, forexample, hardware components such as application specific integratedcircuits (ASICs) and field-programmable gate array (FPGA) devices.Implementation of the hardware state machine so as to perform thefunctions described herein will be apparent to persons skilled in therelevant art.

Additionally, the present invention may be conveniently implementedusing one or more conventional general purpose or specialized digitalcomputer, computing device, machine, or microprocessor, including one ormore processors, memory and/or computer readable storage mediaprogrammed according to the teachings of the present disclosure.Appropriate software coding can readily be prepared by skilledprogrammers based on the teachings of the present disclosure, as will beapparent to those skilled in the software art.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example, and notlimitation. It will be apparent to persons skilled in the relevant artthat various changes in form and detail can be made therein withoutdeparting from the spirit and scope of the invention.

The present invention has been described above with the aid offunctional building blocks illustrating the performance of specifiedfunctions and relationships thereof. The boundaries of these functionalbuilding blocks have often been arbitrarily defined herein for theconvenience of the description. Alternate boundaries can be defined solong as the specified functions and relationships thereof areappropriately performed. Any such alternate boundaries are thus withinthe scope and spirit of the invention.

The foregoing description has been provided for the purposes ofillustration and description. It is not intended to be exhaustive or tolimit the invention to the precise forms disclosed. The breadth andscope should not be limited by any of the above-described exemplaryembodiments. Many modifications and variations will be apparent to thepractitioner skilled in the art. The modifications and variationsinclude any relevant combination of the disclosed features. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical application, therebyenabling others skilled in the art to understand the invention forvarious embodiments and with various modifications that are suited tothe particular use contemplated. It is intended that the scope of theinvention be defined by the following claims and their equivalence.

In the various embodiments described above, unless specifically notedotherwise, disjunctive language such as the phrase “at least one of A,B, or C,” is intended to be understood to mean either A, B, or C, or anycombination thereof (e.g., A, B, and/or C). As such, disjunctivelanguage is not intended to, nor should it be understood to, imply thata given embodiment requires at least one of A, at least one of B, or atleast one of C to each be present.

What is claimed is:
 1. A system for real-time mapping in a movableobject environment, comprising: at least one movable object, including acomputing device; a scanning sensor electronically coupled to thecomputing device; a positioning sensor electronically coupled to thecomputing device; a client device in communication with the at least onemovable object, the client device including at least one processor and avisualization application, the visualization application includinginstructions which, when executed by the processor, cause thevisualization application to: receive point cloud data from the scanningsensor and position data from the positioning sensor; determine aresolution for a real-time visualization of the point cloud data;generate the real-time visualization of the point cloud data and theposition data as it is received based on the resolution; and display thereal-time visualization using a user interface provided by thevisualization application.
 2. The system of claim 1, wherein todetermine the resolution for the real-time visualization of the pointcloud data, the instructions, when executed by the processor, furthercause the visualization application to: determine the resolution basedon available computing resources.
 3. The system of claim 1, wherein todetermine the resolution for the real-time visualization of the pointcloud data, the instructions, when executed by the processor, furthercause the visualization application to: determine the resolution basedon a user input specifying a voxel size.
 4. The system of claim 3,wherein the voxel size is specified via a user interface elementrendered in the user interface which enables the user to incrementallyincrease or decrease the voxel size.
 5. The system of claim 1, whereinthe instructions to generate the real-time visualization of the pointcloud data and the position data as it is received, when executed by theprocessor, further cause the visualization application to: generate thereal-time visualization of the point cloud data based on an intensityvalue associated with each point in the point cloud data.
 6. The systemof claim 5, wherein the intensity value represents a materialcomposition or height above a reference plane associated with each pointin the point cloud data.
 7. The system of claim 1, wherein theinstructions, when executed by the processor, further cause thevisualization application to: obtain sensor data from one or moresensors of the movable object; generate a representation of the sensordata using at least one associated graphical user interface element; anddisplay the representation of the sensor data as an overlay on thereal-time visualization.
 8. A method for real-time mapping in a movableobject environment, comprising: receiving point cloud data from ascanning sensor and position data from a positioning sensor; determininga resolution for a real-time visualization of the point cloud data;generating the real-time visualization of the point cloud data and theposition data as it is received based on the resolution; and displayingthe real-time visualization using a user interface provided by avisualization application.
 9. The method of claim 8, wherein determiningthe resolution for the real-time visualization of the point cloud datafurther comprises: determining the resolution based on availablecomputing resources.
 10. The method of claim 8, wherein determining theresolution for the real-time visualization of the point cloud datafurther comprises: determining the resolution based on a user inputspecifying a voxel size.
 11. The method of claim 10, wherein the voxelsize is specified via a user interface element rendered in the userinterface which enables the user to incrementally increase or decreasethe voxel size.
 12. The method of claim 8, wherein generating thereal-time visualization of the point cloud data and the position data asit is received, further comprises: generating the real-timevisualization of the point cloud data based on an intensity valueassociated with each point in the point cloud data.
 13. The method ofclaim 12, wherein the intensity value represents a material compositionor height above a reference plane associated with each point in thepoint cloud data.
 14. The method of claim 8, further comprising:obtaining sensor data from one or more sensors of a movable object;generating a representation of the sensor data using at least oneassociated graphical user interface element; and displaying therepresentation of the sensor data as an overlay on the real-timevisualization.
 15. A non-transitory computer readable storage mediumincluding instructions stored thereon which, when executed by at leastone processor, cause the at least one processor to perform the steps of:receiving point cloud data from a scanning sensor and position data froma positioning sensor; determining a resolution for a real-timevisualization of the point cloud data; generating the real-timevisualization of the point cloud data and the position data as it isreceived based on the resolution; and displaying the real-timevisualization using a user interface provided by a visualizationapplication.
 16. The non-transitory computer readable storage medium ofclaim 15, wherein to determine the resolution for the real-timevisualization of the point cloud data, the instructions, when executedby the processor, further cause the at least one processor to performthe steps of: determining the resolution based on available computingresources.
 17. The non-transitory computer readable storage medium ofclaim 15, wherein to determine the resolution for the real-timevisualization of the point cloud data, the instructions, when executedby the processor, further cause the at least one processor to performthe steps of: determining the resolution based on a user inputspecifying a voxel size.
 18. The non-transitory computer readablestorage medium of claim 17, wherein the voxel size is specified via auser interface element rendered in the user interface which enables theuser to incrementally increase or decrease the voxel size.
 19. Thenon-transitory computer readable storage medium of claim 15, wherein thestep of generating the real-time visualization of the point cloud dataand the position data as it is received, when executed, further causesthe at least one processor to perform the step of: generating thereal-time visualization of the point cloud data based on an intensityvalue associated with each point in the point cloud data, wherein theintensity value represents a material composition or height above areference plane associated with each point in the point cloud data. 20.The non-transitory computer readable storage medium of claim 15, whereinthe instructions, when executed, further cause the at least oneprocessor to perform the steps of: obtaining sensor data from one ormore sensors of a movable object; generating a representation of thesensor data using at least one associated graphical user interfaceelement; and displaying the representation of the sensor data as anoverlay on the real-time visualization.